Combining Performance Actions with Spectral Models for Violin Sound Transformation

In this work we present a violin timbre model that takes into account performance gestures. It is built by analysis of performance data using machine learning methods and it is able to predict the timbre given a set of performance actions. Gestural data and sound are synchronously captured by means of 3D motion trackers attached to the instrument and a bridge pickup. The model is used for sample transformation within a spectral concatenative synthesizer informed by gestures. INTRODUCTION Spectral concatenative synthesis models [1], [2] generate sound by concatenation of spectrally transformed samples. Sample concatenation is crucial for the quality of sound produced, and sometimes transitions between two samples do not sound natural, especially in the case of sustained excitation instruments such as the violin, because they have a wider timbre space and need a continuous control. One manner in which to improve these models in order to provide better controllability and expressive capabilities is to take into consideration performance gestures, that is, informing the model with "how is the instrument played". Performance actions are sound producing gestures articulated by the musician that control/drive the production of sound (see in [8] for a categorization of musical gestures). When performing with a violin, one can produce a wide range of different timbre variations, by applying a complex combination of actions controlled by bow and fingers. Bowing actions are the most relevant concerning timbre and therefore we will focus on them. We have developed a sensing system [14] by making use of two Polhemus 3D-motion trackers. Using data provided by this system we obtain bowing performance actions with great accuracy. Sound is acquired by means of a 4-channel bridge pickup that is then spectrally analyzed. With this setup, we are able to synchronously collect large amounts of performance data (gestures and sound), that is used to train a set of neural networks. The trained networks are finally used in the transformation stage of a spectral concatenative synthesizer. The paper is structured as follows: First we describe which data is acquired and how. In the case of sound recording, we discuss why are we using a bridge pickup instead of another device. Then we present the neural network that models the timbre, detailing its structure, inputs, output, the dataset for training and its performance. Finally we outline its use in the transformation procedure by the synthesizer and conclude by commenting some evaluation results and presenting further developments of the model. DATA AQUISITON With our measuring system consisting of the bridge pickup and the motion trackers we can capture an enormous amount of performance data. The main advantages over other systems like bowing machines [4] are that the range of the bowing actions is not constrained by the machine and that we can capture real performance data.